Applying Rule-Based Normalization to Different Types of Historical Texts - An Evaluation

This paper deals with normalization of language data from Early New High German. We describe an unsupervised, rule-based approach which maps historical wordforms to modern wordforms. Rules are specified in the form of context-aware rewrite rules that apply to sequences of characters. They are derived from two aligned versions of the Luther bible and weighted according to their frequency. Applying the normalization rules to texts by Luther results in 91% exact matches, clearly outperforming the baseline (65%). Matches can be improved to 93% by combining the approach with a word substitution list. If applied to more diverse language data from roughly the same period, performance goes down to 42% exact matches (baseline: 32%), but is higher than using a wordlist. The results show that rules derived from a highly different type of text can support normalization to a certain extent.